To reconcile the conflicts, we propose a robust federated learning framework against poisoning attacks (RFLP A) based on SecAgg protocol.

In this work, we investigate how an adversary can exploit instruction tuning by injecting specific instruction-following examples into the training data that intentionally changes the model's behavior.

Earlier research highlighted DMs' vulnerability todatapoisoning attacks, butthese studies placed stricter requirements than conventional methods like'BadNets' inimage classification.

DDFedsimultaneously boosts privacyprotection andmitigatespoisoning attacks, without introducing new participant roles or disrupting the existing FL topology.DDFedinitially leveragescutting-edge fullyhomomorphic encryption (FHE)tosecurely aggregatemodelupdates, without theimpractical requirement for non-colluding two-server setups and ensures strong privacy protection.

Data poisoning considers cases when an adversary manipulates the behavior of machine learning algorithms through malicious training data.

This paradigm has been challenged by practical deployments, which are more dynamic in nature.

The first is a traditional Label Attack, tricking VLMs into misidentifying class labels, such as confusing Donald Trump for Joe Biden.